Manufacturers of Communications devices (e.g., routers, PBX, communications servers, communications platforms) and application developers have adopted new open standards protocols such as TAPI, JTAPI, S.100 and CSTA, or developed their own proprietary protocols. As new products implementing these protocols are delivered to the marketplace, the need for device and software manufacturers to test for conformance to standards, performance and stability, as well as interoperability with applications, is increasingly critical. When testing complex communications and telephony systems, different types of tests must be performed to simulate all of the possible conditions and states the target system encounters during “live” operation. This variety of testing ensures that a telecommunications or computer telephony system is ready to provide a high-level of quality service. This testing also ensures that the target system continues to provide appropriate service during full operation. It is desirable for a testing system to be able to perform each of the following types of testing:
FUNCTIONALITY AND CONFORMANCE TESTING—The conformance testing refers to whether an application/system performs according to the related specifications. Manufacturers must ensure that their products are interoperable with applications that implement the same claimed standards. For example, a TAPI 3.X service provider should work with all the TAPI 3.X applications on the market and a TAPI 3.X application should work with all the TAPI 3.X service providers. In addition, interoperability between multiple applications sharing the same resources is critical. For example, with regard to application programming interfaces, it is important to ensure compatibility among applications from different vendors that perform distinct functions (e.g., call control, billing, etc) that share the same resources and which are integrated into a single solution. The service provider and the application need to be tested as mission critical software in order to offer the needed reliability and stability. To achieve this goal, testing must be done at both the service provider and application level.
LOAD AND STRESS TESTING—Telecommunications systems break down most frequently due to intensive usage or “load” situations. As a result, it is important to load the telecommunications system using real-world traffic, which simulates the actions, usage patterns, and media types (e.g. voice, fax, data, video) of real users during busy hours. Load tests should also analyze the responses that are returned from the target system to ensure that they are completely accurate. If system degradation or failure occurs, it must be determined why the problem occurred and it is necessary to pinpoint exactly what caused the problem in the target system.
ENDURANCE TESTING—Endurance testing measures the capability of the system under test to preserve its designed functionality when working in real world conditions over a long period of time.
REGRESSION TESTING—Hardware and software for telecommunications systems are upgraded periodically to add new system features or correct existing bugs. Whenever these types of changes are made to complex software systems, a tremendous risk exists because these changes or additions to the target system could corrupt its existing features. This problem requires the comprehensive testing of all the related features of the target system. Due to the amount of time required to manually perform these tests, it is critical to automate this regression testing process.
Various systems for performing communications testing are known in the art, such as the ChannelGauge system from G3 Nova Technologies, Inc. of West Lake Village, Calif., the “Hammer” line of testing systems from Emprix, Inc. of Waltham, Mass., and the Abacus and Abacus 2 testing systems from Zarak Systems of Sunnyvale, Calif. However, each of these systems has shortcomings that preclude the type of thorough and easy to use testing system of the present invention. For example, it is known to provide software to create Test Scenarios using a graphical user interface, in which different Test Functions are associated with different Communications Channels. However, this alone does not provide synchronization of Test Functions across multiple Communications Channels with different Interface Protocols, or the ability to incorporate new Interface Protocols, such as telephony applications programming Interface Protocols, into the scenario builder. Furthermore, it is known to scan a network to identify computers or agents that may be utilized in a testing scenario, but this does not provide an ability to itemize the Interface Protocols that may be used on each computer. Moreover prior systems do not provide for analysis profiles, cross analysis or layered result analysis of test results. It is also known to provide a communications system tester that uses telephony API function calls to control a communications platform via the communications server software that directly controls the communications hardware of the communications platform. Such functionality is present in the TAPI Capacity Tester from Genoa Technology, of Moorpark, Calif. 93021. However, many of the foregoing systems suffer from the deficiency of not being able to provide a testing system that can simultaneously test all of the Interface Protocols utilized by today's complex communications platforms. This, in part, is due to multitude of ways that Communications Channel endpoints are identified and controlled in a complex communications platform. For example, the information needed to control a Voice over IP (VoIP) Communications Channel endpoint might be a H323 endpoint, an IP address, and a unique phone number identifier. However, the information needed to control a T1 Communications Channel that terminates in a Dialogic board, for example, might be the Dialogic board name, the span, timeslot, and a unique phone number identifier. The information needed to control a CDR Communications Channel might be a CTM Server name and a computer name. Likewise, for the TAPI 2.x Interface Protocol, the information needed to identify and control a specific Communications Channel endpoint could include a Line ID, and address, and line information. Accordingly, as the number of Interface Protocols for a communications platform increase, the difficulty of providing a system that can simultaneously control and test different Communications Channels that use different Interface Protocols of the communication platform increases exponentially.
Many other problems relating to testing are caused by the increased complexity of communications platforms. For example, because newer communications platforms have greater capacity, it is necessary to utilize more test equipment (such an automated bulk call generators) to interface with a communications platform to test it. However, it is still important to have a central console from which the Test Scenario is controlled. Because the central console must be networked to the test equipment during the test, this means that there will be significant network traffic between the testing console and the test equipment. This network traffic devoted to executing the Test Scenario, and collecting the resulting data, can itself become a bottleneck, which prevents the test equipment from being utilized to its capacity for testing the communications platform. Accordingly, there is a need for a system that reduces network traffic associated with the testing console and test equipment.
This problem has been partially addressed by systems such as ChannelGauge of G3 Nova Technologies, Inc. of Westlake Village, Calif. This product uses Agents, or software, usually on remote computers, connected to test equipment, that can receive all or a portion of a Test Scenario and independently execute it and without the necessity of significant overhead network traffic from the testing console. In addition, Agents record command responses and status information from communications devices regarding a Communications Channel. However, the use of Agents creates additional problems. It is more difficult to define a Test Scenario on the testing console when there a multiple Agent computers. It is difficult to keep track of which computers are connected to which test equipment, and the communications resources of each test piece of test equipment connected to each Agent computer. In addition, as communications platforms increase in complexity by implementing more Interface Protocols, it will become even more difficult to keep track of which Agents are interfaced with test equipment that implements the various Interface Protocols. While all of this information can be “hard coded” into a Test Scenario, it makes the Test Scenarios more difficult to write and modify. Moreover, the use of Agents does not necessarily diminish the network traffic associated with transmitting test results to the testing console (or a separate database server).
Another problem resulting from the increased capacity of communications platforms having many Communications Channels is the difficulty of writing Test Scenarios to test all of the Communications Channels. Writing Test Scenarios is a tedious process, and it is made worse by communications platforms having hundreds or even thousands of Communications Channels. Accordingly, it is desirable to provide a system and method to increase the efficiency with which Test Scenarios can be written for platforms having large numbers of Communications Channels.
Another significant problem with newer communications platforms is that many are designed to interface with other pieces of expensive network equipment. Sometimes, the other expensive network equipment can cost over a million dollars or more. As a practical matter, many testing facilities do not have the resources to purchase all of the network equipment designed to interact with a communications platform. This can lead to instances where a new communications platform can not be adequately tested before being released to the public, which decreases the reliability of the communications platform. Accordingly, it is desirable to provide a method for emulating communications equipment that a communications platform is intended to interact with so that the platform can be tested without having to purchase expensive equipment.
Another problem with testing communications platforms is that they are increasingly provided by their manufacturers with communications services provider software having a telephony API that is not generally accepted in the marketplace, or even worse, proprietary. This makes it difficult to thoroughly test the communications services provider software, or other devices intended to connect to interact with it.
Another problem with communications platform test systems is the ability to test platforms that implement multiple Interface Protocols. Generally, some systems exist that allow, for example, testing of multiple telephony applications program interfaces, or which allow the testing of internet protocol communications or TDM Interface Protocol communications, but not in a simultaneous Test Scenario. These limits the “real world” testing that can be done, as it is desirable for a platform to be testing all Interface Protocols simultaneously. Moreover, to conduct Test Scenarios, it is often desirable to synchronize the Test Functions that take place across different Communications Channels. For example, it is often necessary to place a call on a first Communications Channel to a second Communications Channel, and the have the call answered on the second Communications Channel. While synchronization has been implemented in test systems in which both Communications Channels use the same Interface Protocol, synchronization has not been effectively implemented across different Interface Protocols, such as when a call is placed via a TDM Communications Channel to a voice of Internet Protocol (VoIP) phone.
A more fundamental problem with communications testing systems relates to their flexibility to keep up with the pace of new Interface Protocols for new communications platforms. It is very difficult and time-consuming to write a testing system for a specific Interface Protocol, yet new Interface Protocols are introduced on the market with increasing frequency. This has required significant rewriting of communications testing software systems, which is slow and expensive. Accordingly it is desirable to provide a communications testing system that allows new Interface Protocols to be “plugged in” so they can be used without significant rewriting of the core testing software.
Another problem with communications testing systems is the ability to provide meaningful results. It is possible to capture a great deal of information about each Test Function in a Test Scenario, such as the exact time the function began, the command used to implement the function, the result code, the Communications Channel status, etc. However, this can result in information overload when many iterations of tests comprised of numerous Test Functions are run across thousands of Communications Channels. At times, highly detailed information is necessary to pinpoint equipment design flaws and bottlenecks, but when testing only for capacity specifications, collecting too much information is undesirable, and as noted above, the excessive collection of information can actually impede the ability to perform the tests. Accordingly, it is desirable to provide a communications testing system with flexible information collection and analysis capabilities. In particular, it is desirable to generate information that provides various levels of detail, and allows visual display so that it may be easily understood. In addition, it is desirable to provide a system that allows pass/warning/fail criteria to be specified after a test is complete and the data has been gathered, so that the performance of a communications platform may be more completely assessed.
Another problem with communications test systems relates to the ability to obtain information regarding the status of a Communications Channel. For example, different telephony APIs (such as TAPI, JTAPI, S.100, S.410, CDR and WMI) provide different levels of information regarding the status of a Communications Channel. For example, a communications service provider for a communications platform can support multiple both S.100 and TAPI telephony API. The S.100 telephony API provides detailed information about a streaming media communication, such as a playing wave files or tone files, or bridging conversations. However, the TAPI telephony API provides more powerful call control processing. Accordingly, it would be desirable to have a communications test system that can use multiple protocols to obtain the maximum information about the Communications Channel of a communications platform.